Enable running PyTorch models #207
Conversation
| block_tables.append(block_table.get_blocks()) | ||
|
|
||
| if sliding_window: | ||
| seq_lens.append(min(seq_len, sliding_window)) | ||
| else: | ||
| seq_lens.append(seq_len) | ||
|
|
||
| max_context_len = max(max_context_len, seq_lens[-1]) | ||
|
|
||
| def _do_pad( |
There was a problem hiding this comment.
Now that we started considering vllm's tensor layout, what do you think about unifying it? It seems like upstream mlc-llm also uses 2D inputs.
There was a problem hiding this comment.
And this also could help our cuda graph integration?
There was a problem hiding this comment.
We haven't verified if 2D inputs is better for performance, and how much cuda graph actually helps.
The upstream input looks like 2D but it is always either (1, num_total_token) or (batch_size, 1). So their 2D input is essentially 1D.
There was a problem hiding this comment.
Yeah, I think it is worth visiting imo. But not now, in the future. Although there might not be performance boost, it would be nice to unify the layout with upstream unless there is reason.
|
Oh I just realized that, after I updated our vLLM work, which made their input representation 2D, memory profiling is broken. Right now we only have We could look into how vLLM does memory profiling currently, but I'm inclined to revert their 2D rep change instead. Either way, this is left for future work. |
* Prioritize arch-specific lib * Avoid repetitive dlopen when reloading the same model * handle the case where lib path remains the same
This is the first cut toward running PT models under
mlc-serve. Things are functional but rough. All we need to do to run a PT model is to replace--local-id llama-2-13b-chat-hf-q0f16with--local-id models/llama-2-13b-chat-hf(the path to the HF model directory) etc.from vllm.model_executor.models.llama import LlamaForCausalLMetc. Models such as phi-2, Qwen run out of the box, in addition to llama-based models and Mixtral that the TVM model also supports. vLLM needs to be installed from https://github.com/octoml/vllm/tree/for-mlc-servebenchmark_throughput.py, TVM gets22.98 requests/swhile PT gets13.78 requests/s. Since TVM and vLLM models should be using the same kernels for matmul and attention, there shouldn't be a big difference in performance. I haven't looked into this issue deeply.pollbetween two decode).@sunggg @yelite @elvin-n @vvchernov @jroesch @binarybana